关键词: 连接算法, 数据倾斜, 全局分区, 采样

Abstract: The study of join algorithm based on MapReduce is a hot topic in massive data research area. However, most current optimization work is based on the assumption that the data are evenly distributed. In practical applications, the data to be processed are often skew in distribution. This paper proposes a MapReduce join algorithm called Skew Control Join, which is adaptive for serious skew data. The algorithm gets the overall data distribution by sampling, then partitions the data by total partitioner to distribute the data evenly to all Reduce tasks. Experiment results show that the algorithm is of good performance when the processed data are skew.

Key words: join algorithm, data skew, total partition, sample